An Efficient Algorithm for Outlier Detection in High Dimensional Real Databases

Detecting outlier patterns in data has been an important research topic in statistics, data mining and machine learning communities for many years. Research in identifying effective solutions to this problem have several interesting applications in a myriad of domains ranging from data cleaning to financial fraud detection and from network intrusion detection to clinical diagnosis of diseases. Among the different algorithms, statistical (parametric) approaches and distance-based outlier detection are the most popular in use. The former is well grounded but often has difficulty scaling to large and high dimensional data. The latter is relatively efficient and empirically found to be effective on a number of domains but scalability is still an issue in spite of a fair bit of research on the topic. To address this limitation, in this work, we propose Atalaia, an efficient and scalable distance-based algorithm for detecting outliers in large high dimensional databases. Central to our algorithm is a fast strategy to estimate the unusualness of a record within the database and use a rank-ordered approach to evaluate records. Our algorithm partitions the database and ranks the objects that are candidates to be an outlier, reducing significantly the number of comparisons among objects. We evaluate different ranking heuristics in a comprehensive set of real and synthetic databases. Further, Atalaia also handles heterogeneous databases, i.e, those containing both categorical and continuous attributes. The results show that our algorithm outperforms by up to 73% the state-of-the-art distance-based outlier detection algorithm.